What do these changes do

Adds comprehensive type annotations to all files in pythainlp.tokenize, achieving 100% type coverage verified by mypy. Implements TYPE_CHECKING pattern for nlpo3 to preserve type information with lazy imports, and uses "import as" pattern to avoid type: ignore comments for import shadowing.

What was wrong

The pythainlp.tokenize submodule lacked complete type hints, preventing static type checking and IDE intelligence. Specific issues included:

• Missing return type annotations in several functions
• nlpo3 >= 1.4.0 provides type annotations but module-level imports would fail if nlpo3 wasn't installed
• Multiple conditional imports of segment functions caused import shadowing, requiring # type: ignore[assignment] suppressions
• Incorrect return types in word_detokenize and sent_tokenize
• External libraries (attacut, deepcut, oskut, sefr_cut, budoux, wtsplit, ssg, tltk) return Any without type information

How this fixes it

1. TYPE_CHECKING pattern for nlpo3:
Uses Python's typing.TYPE_CHECKING to separate type-time imports from runtime imports:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from nlpo3 import segment as nlpo3_segment  # noqa: F401

def segment(text: str) -> list[str]:
    try:
        from nlpo3 import segment as nlpo3_segment
    except ImportError as ex:
        raise ImportError("nlpo3 is not installed. Install it with: pip install nlpo3") from ex
    
    return nlpo3_segment(text=text, ...)

During type checking: mypy sees nlpo3's type information
At runtime: Module imports successfully without nlpo3; lazy import with helpful error message

2. Import as pattern for conditional imports:
Replaced import shadowing with unique aliases:

# Before: from pythainlp.tokenize.sefr_cut import segment  # type: ignore[assignment]
# After:  from pythainlp.tokenize.sefr_cut import segment as sefrcut_segment

This eliminates all # type: ignore[assignment] comments while making code more explicit.

3. Comprehensive type hints:

• Added return types to all functions (longest.py, core.py, _utils.py)
• Fixed return types: word_detokenize → Union[list[list[str]], str], sent_tokenize → Union[list[str], list[list[str]]]
• Updated _utils.py to use Sequence for covariance
• Applied cast() for libraries without type information (attacut, budoux, deepcut, oskut, sefr_cut, wtsplit, ssg)
• Removed unreachable code in crfcut.py and nercut.py

Results:

• ✅ 0 mypy errors in tokenize module (down from 31)
• ✅ nlpo3 removed from mypy ignore list, preserving type information
• ✅ No # type: ignore[assignment] needed for import shadowing
• ✅ All tests pass, Ruff checks pass

Your checklist for this pull request

• Passed code styles and structures
• Passed code linting checks and unit test

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.